Dockable billboards for labeling objects in a display having a three dimensional perspective of a virtual or real environment

ABSTRACT

A system and method that allows a user to view objects in a three-dimensional environment, where one or more of the objects have a data display (e.g., a data billboard, etc.) that shows data about the object. To enhance user experience and to provide relevant contextual data as the user navigates through the three-dimensional environment, the system calculates a location for the user and a location for each object and determines if a relationship between the user frame of reference and each object location satisfies a first criterion. If the first criterion is satisfied, the system is configured to move the data display to the bottom of a viewing area of the three-dimensional environment (e.g. docking the data display to the bottom of the viewing area, etc.). The system may also arrange the data displays in the same order as the objects are perceived by the user in the three-dimensional environment.

PRIORITY CLAIM

This application is a continuation of the U.S. patent applicationtitled, “DOCKABLE BILLBOARDS FOR LABELING OBJECTS IN A DISPLAY HAVING ATHREE-DIMENSIONAL PERSPECTIVE OF A VIRTUAL OR REAL ENVIRONMENT,” filedon Jul. 18, 2019 and having Ser. No. 16/515,429, which is continuationof the U.S. patent application titled, “DOCKABLE BILLBOARDS FOR LABELINGOBJECTS IN A DISPLAY HAVING A THREE-DIMENSIONAL PERSPECTIVE OF A VIRTUALOR REAL ENVIRONMENT”, filed on Apr. 30, 2014 and having Ser. No.14/266,511, now Pat. No. as 10,380,799, issued Aug. 13, 2019, whichclaims the priority benefit of U.S. provisional patent applicationtitled, “ DOCKABLE BILLBOARDS FOR LABELING OBJECTS IN ATHREE-DIMENSIONAL DISPLAY OF A VIRTUAL OR REAL ENVIRONMENT”, filed Jul.31, 2013 and having Ser. No. 61/860,882. The subject matter of theserelated applications is hereby incorporated herein by reference.

TECHNOLOGY

The present invention relates generally to information systems, and inparticular, to extracting and viewing data generated by informationsystems.

BACKGROUND

Information systems generate vast amounts of information that can bedifficult to view and from which it can be difficult to extract usefulinsight. Although the development of computers and software has beenstaggering in many ways, existing computer systems are still limited intheir capacity to convey large amounts of data in a way that users candigest and understand quickly. Because the amount of relevant data thatis available for analysis continues to increase significantly from yearto year, the need for improved tools for communicating such data tousers is becoming urgent.

The approaches described in this section are approaches that could bepursued, but not necessarily approaches that have been previouslyconceived or pursued. Therefore, unless otherwise indicated, it shouldnot be assumed that any of the approaches described in this sectionqualify as prior art merely by virtue of their inclusion in thissection. Similarly, issues identified with respect to one or moreapproaches should not assume to have been recognized in any prior art onthe basis of this section, unless otherwise indicated.

BRIEF DESCRIPTION OF DRAWINGS

The present invention is illustrated by way of example, and not by wayof limitation, in the figures of the accompanying drawings and in whichlike reference numerals refer to similar elements and in which:

FIG. 1 is block diagram of an example data display system that providesdockable billboards for labeling objects in a three-dimensional displayof a virtual or real environment in accordance with an embodiment of thepresent system;

FIG. 2 is a perspective view of an example wearable computer that allowsa user to view a real environment in accordance with an embodiment ofthe present invention;

FIG. 3 is a block diagram of the wearable computer shown in FIG. 2 ;

FIG. 4 comprises a flow diagram illustrating an example processperformed by the system of FIG. 1 or the system of FIG. 3 in accordancewith an embodiment of the present method;

FIG. 5 through FIG. 8 illustrate an example view of a three-dimensionalenvironment as perceived from a virtual camera located within thevirtual environment created by the system of FIG. 1 in accordance withone embodiment of the present system and methods;

FIG. 9 and FIG. 10 illustrate an example view of a real environment asperceived by a camera located in the real environment in accordance withan embodiment of the present systems and methods; and

FIG. 11 is a block diagram of a computer that may be used, for example,as a client device or server computer within the context of the systemof FIG. 1 or as the computer as used in the context of the system ofFIG. 2 .

FIG. 12 illustrates an example process flow.

DESCRIPTION OF EXAMPLE EMBODIMENTS

Example embodiments, which relate to extracting and viewing data, aredescribed herein. In the following description, for the purposes ofexplanation, numerous specific details are set forth in order to providea thorough understanding of the present invention. It will be apparent,however, that the present invention may be practiced without thesespecific details. In other instances, well-known structures and devicesare not described in exhaustive detail, in order to avoid unnecessarilyoccluding, obscuring, or obfuscating the present invention.

Example embodiments are described herein according to the followingoutline:

-   -   1. GENERAL OVERVIEW    -   2. STRUCTURE OVERVIEW    -   3. SECOND EXEMPLARY DATA DISPLAY SYSTEM    -   4. EXEMPLARY DATA DISPLAY METHOD    -   5. EXEMPLARY USER EXPERIENCES    -   6. FIRST EXEMPLARY USER EXPERIENCE    -   7. SECOND EXEMPLARY USER EXPERIENCE    -   8. EXAMPLE PROCESS FLOW    -   9. EXEMPLARY COMPUTER ARCHITECTURE    -   10. EQUIVALENTS, EXTENSIONS, ALTERNATIVES AND MISCELLANEOUS

1. General Overview

This overview presents a basic description of some aspects ofembodiment(s) of the present invention. It should be noted that thisoverview is not an extensive or exhaustive summary of aspects of theembodiment. Moreover, it should be noted that this overview is notintended to be understood as identifying any particularly significantaspects or elements of the embodiment(s), nor as delineating any scopeof the embodiment(s) in particular, nor the invention in general. Thisoverview merely presents some concepts that relate to exampleembodiments in a condensed and simplified format, and should beunderstood as merely a conceptual prelude to a more detailed descriptionof example embodiments that follows below.

In various embodiments, a system and method is provided that allows auser to view objects in a three-dimensional environment (e.g., a 3Dspace, a 3D spatial region, etc.) with a display (e.g., a wearabledisplay, on a two or three-dimensional display, or on any other suitabledisplay, etc.), where one or more of the objects have a data display(e.g., a billboard, etc.) associated with the object that shows dataabout the object. In various embodiments, the data display containsreal-time streaming textual data about the object. To enhance the userexperience as the user navigates through the three-dimensionalenvironment, the system calculates a location of the user and a locationfor at least one object. The system then determines a distance betweenthe user and the at least one object. If the distance between thelocation of the user and the at least one object is within apredetermined distance, the system may be configured to move the datadisplay for the at least one object to a designated area (e.g., awayfrom the at least one object, etc.) such as the bottom of a viewing area(which may be a part or the entire display area of a display/screen of acomputing device) of the three-dimensional environment (e.g., dockingthe data display at the bottom of the viewing area). When more than onedata display is docked at the bottom of the viewing area, the system maybe configured to arrange the data displays in the same order as theobjects are perceived by or displayed to the user.

In some embodiments, the three-dimensional environment is a computergenerated virtual environment. The three-dimensional environment can beviewed by the user on a two-dimensional or three-dimensional displayfrom the perspective of a virtual camera that is positioned in theenvironment. Thus, a displayed view of the three-dimensional environmentshown on the display is determined by the location and the perspectiveof the virtual camera. As the user navigates through the virtualenvironment (e.g., moves the location of the virtual camera in thevirtual environment, etc.), for example by using a joystick, acontroller, a mouse, a keyboard, or any other suitable method ofcontrolling the location and direction of the virtual camera, the datadisplays may be docked or undocked at the bottom of the viewing area,and/or arranged based on the arrangement of the objects in displayedviews of the three-dimensional environment as perceived from the virtualcamera. Docking of the data displays may be carried out when thelocation of an object with respect to the location of the virtual camerasatisfies a data display docking criterion (e.g., the distance betweenthe object and virtual camera is less than a threshold distance, theobject is located within a predetermined radius from the location of thevirtual camera, any other suitable criterion, etc.). Additionally,optionally or alternatively, the data display docking criterion is basedat least in part on a function, algorithm, etc., that depends on one ormore of (a) a distance of a three-dimensional object relative to theuser's location as represented in the three-dimensional environment, and(b) an angle between (1) a first imaginary line connecting athree-dimensional object (e.g., the central location of thethree-dimensional object, etc.) and the user's location, and (2) asecond imaginary line representing the user's perspective. Additionally,optionally or alternatively, a user or a system can specificallydesignate/configure a three-dimensional object to have its correspondingdata display to be docked in a designated area of a view of thethree-dimensional environment; for example, the configurable designationof the three-dimensional object to have its corresponding data displayto be docked may be included as a part of the data docking criteria.

In various other embodiments where the three-dimensional environment isa superimposition of (1) a computer-generated three dimensionalenvironment that comprises graphic objects, data displays, etc., and (2)a real-world environment that surrounds the user, the three-dimensionalenvironment may be viewed by the user through a wearable display such asa heads-up display, display glasses, etc., which includes, but is notlimited to only, a display attached to a wearable computer that isimaging the real-world environment in real-time as the user movesaround. In these embodiments, the system may receive real-time data,non-real-time data, etc., about objects in the surrounding environmentand overlay/superimpose at least a portion of the data on the objectsviewed by the user or imaged by a camera. For example, if a user iswearing GOOGLE GLASSES®, the user can walk through New York City andview the buildings, shops, and other objects surrounding the user as theuser walks down 5th Avenue. Based on position and location informationdetermined by the GOOGLE GLASSES® using various sensors (e.g., GPS,motion sensors, head-tracking sensors, etc.), the system may downloadinformation about the various buildings, shops, and landmarks (e.g.,objects, etc.) and display the information on data displays (e.g., abillboard, etc.) associated with the object(s) viewed by the user. Thus,as the user navigates down 5th Avenue, the system can move (e.g., dock,etc.) various billboards for objects that satisfy a criterion (e.g., theobject is within a threshold distance from the user, etc.) to the bottomof the viewing area (as defined by the heads-up display or display) inthe glasses. This allows the user to focus on billboards for objectsthat are relevant to the position of the user in the user's surroundingenvironment. In some embodiments, while a user is walking in areal-world environment, a wearable computing device may render 2D and 3Ddisplayed graphic objects (e.g., undocked billboards, markers, shapes,maps, etc.) at specific coordinates (e.g., x-y-z coordinates of a spacerepresenting the user's real environment, etc.) of the user's real-worldenvironment, for example, as if the 2D and 3D graphic objects, etc., area part of the user's real-world environment.

In still other embodiments where the three-dimensional environment is athree-dimensional interactive map (e.g., APPLE® 3D with navigationfunctionality, etc.) a user may view a map of an area on a handhelddevice, a tablet or any other display device. As the user navigatesthrough the map, the system may be configured to overlay data billboardson one or more of the objects (e.g., buildings, landmarks, monuments,restaurants, etc.) in the displayed view to provide the user withrelevant data associated with the objects. In this way and similar tothe embodiments described above, as the virtual camera viewpoint ismoved through the 3D map, the user can see information about buildings,landmarks, monuments, retail establishments and any other objects beingdisplayed to the user. Additionally, the data displays for objectswithin a predefined proximity to the virtual camera location may bedocked at the bottom of the display so that relevant information aboutone or more objects within the predefined proximity can be easily viewedby the user.

Various modifications to the preferred embodiments and the genericprinciples and features described herein will be readily apparent tothose skilled in the art. Thus, the disclosure is not intended to belimited to the embodiments shown, but is to be accorded the widest scopeconsistent with the principles and features described herein.

2. Structure Overview

As will be appreciated by one skilled in the relevant field, the presentsystems and methods may be, for example, embodied as a computer system,a method, or a computer program product. Accordingly, variousembodiments may be entirely hardware or a combination of hardware andsoftware. Furthermore, particular embodiments may take the form of acomputer program product stored on a computer-readable storage mediumhaving computer-readable instructions (e.g., software, etc.) embodied inthe storage medium. Various embodiments may also take the form ofinternet-based computer software and/or hardware. Any suitablecomputer-readable storage medium may be utilized including, for example,hard disks, compact disks, DVDs, optical storage devices, and/ormagnetic storage devices.

Various embodiments are described below with reference to block diagramsand flowchart illustrations of methods, apparatuses (e.g., systems,etc.), and computer program products. It should be understood that eachblock of the block diagrams and flowchart illustrations, andcombinations of blocks in the block diagrams and flowchartillustrations, respectively, can be implemented by a computer executingcomputer program instructions. These computer program instructions maybe loaded onto a general purpose computer, a special purpose computer,or other programmable data processing apparatus to produce a machine. Assuch, the instructions which execute on the general purpose computer,special purpose computer, or other programmable data processingapparatus can direct a computer or other programmable data processingapparatus to function in a particular manner such that the instructionsstored in the computer-readable memory produce an article of manufacturethat is configured for implementing the functions specified in theflowchart block or blocks. The program code may execute entirely on theuser's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer, or entirely on the remote computer or server. In the latterscenario, the remote computer may be connected to the user's computerthrough any type of network, including: a local area network (LAN); awide area network (WAN); a cellular network; or the connection may bemade to an external computer (for example, through the Internet using anInternet Service Provider).

FIG. 1 shows a block diagram illustrating an example data display systemfor displaying dockable billboards for labeling objects in athree-dimensional display of a virtual or real environment. The systemcomprises a server (or computing device) 115 that communicates with aplurality of data sources 105 and a plurality of client devices 140 overa network 110 (e.g., the Internet, a cellular network, Wi-Fi, etc.). Forthe purpose of illustration only, as depicted in FIG. 1 , a dockabledata display module 120 is included in server 115. However, it should benoted that in various embodiments, one or more dockable data displaymodules (e.g., 120, etc.) may be located in one or more of clients(e.g., client device 140, etc.), servers (e.g., 115, etc.), etc. Forexample, a client device (e.g., 140, etc.) can comprise a dockable datadisplay module configured to communicate with a plurality of datasources 105 over network 110.

In various embodiments, the data sources 105 may include any source ofdata that is relevant to objects in the three-dimensional environment orreal-world environment. For example, data source 105 may includecomputers, routers, databases, operating systems, applications, mapdata, Internet data or any other source of data. For example, in variousembodiments where one or more data sources are computers, each datasource 105 may produce one or more different types of machine data (e.g.structured data, unstructured data, unparsed data, server logs, activitylogs, configuration files, messages, database records, and the like). Inother embodiments where a data source 105 provides map data, one datasource may supply 3-dimensional, interactive map data while another datasource 105 may supply information about various objects located withinthe map.

In various embodiments, the server may receive data from data sources105 over network 110 in real-time, which may be stored in random accessmemory but not stored locally at server 115. In other embodiments, theserver 115 may receive the data from the data sources 105 and store thedata locally at server 115, for later use. In these embodiments, thedata and three-dimensional environment may be transmitted from server115 to one or more of the client devices 140 for viewing by a user. Instill other embodiments, the data may be transmitted from the server 115to the client device where it is either immediately viewed or storedlocally for later use.

It should be understood that FIG. 1 is an example embodiment of thepresent system and various other configurations are within the scope ofthe present disclosure. Additionally, it should be understood thatadditional devices may be included in the system shown in FIG. 1 , or inother embodiments, certain devices may perform the operation of otherdevices shown in the figure. For purposes of this disclosure, referenceto a server, a computer, or processor, shall be interpreted to include:a single server, a single processor or a single computer; multipleservers; multiple processors, or multiple computers; or any combinationof servers and processors. Thus, while only a single server isillustrated, the term “computer”, “server”, and “processor” may alsoinclude any collection of computers, servers or processors thatindividually or jointly execute a set (or multiple sets) of instructionsto perform any one or more of the methodologies discussed herein.

3. Second Exemplary Data Display System

FIG. 2 shows a perspective view of a wearable computer 200 (e.g., aclient device 140 of FIG. 1 , etc.) with an optical head mounted display202 coupled to a frame 204 having first and second temples 206 and 208and nose supports 210. The wearable computer 200 displays information ina smartphone like, hands-free format that can interact with the Internetand other application via natural language voice commands. The wearablecomputer 200 has a camera 212 that is capable of taking photos andrecording video. The wearable computer may contain various sensorsincluding, but not limited to, gyroscopes, accelerometers,magnetometers, ambient light and proximity sensors, geosynchronouspositioning sensors, a geomagnetic field sensor, an orientation sensorand gravity sensors (further described with reference to FIG. 3 ), etc.The wearable computer may also have one or more of a Wi-Fi transceiver,a Bluetooth transceiver, a cellular transceiver (or a single device thatfunctions as all of these communication devices), storage memory, randomaccess memory, a processor (further described with reference to FIG. 3), etc., which may be contained in one of the housings 214 a and 214 b.In various embodiments, the wearable computer may contain one or moreheads-up displays—instead of or in addition to the optical display202—that is mounted adjacent to, or unitarily formed with lenses mountedin frame 204. The one or more heads-up displays allow the wearer (oruser) to view a scene in the surrounding environment where data, eitherstored in memory or streamed real-time, to the wearable computer isprojected on the one or more heads-up displays so as to align with,superimpose/overlay or be linked to, at least one object viewed by theuser.

In various embodiments, the wearable computer 200 may take the form ofremovable modular housings 214 a and 214 b that couple to the frame 204.The housings 214 a and 214 b may be removably secured to a temple (e.g.,the temple 206, etc.) of the frame 204. In other embodiments, thewearable computer 200 may be integrally formed with the frame 204. Frame204 may include a pair of lens openings (not shown) for mountingprescription or non-prescription lenses. In other embodiments as shownin FIG. 2 , no lens openings are present. The lens openings may definean optical path for a user wearing the frame 204.

FIG. 3 is a block diagram illustrating the components of the wearablecomputer shown in FIG. 2 . One of the housings 214 a and 214 b maycontain and/or support a circuit 216, a speaker 218 for providing soundto the user, a microphone 220 for receiving voice commands or speechfrom the user, a haptic device 222 for providing input to the user, abone conduction device 224 for providing vibration to the user, and atleast two accelerometers 226 a-226 b that are used by the electronics/RFcircuit 216 to receive inputs from the user. Furthermore, one of thehousings 214 a and 214 b may also contain one or more batteries 228 thatprovide power to the circuit 216. The one or more batteries 228 may beremovably or permanently disposed within the one of the housings 214 aand 214 b.

The circuit 216 comprises a processor 236, a transceiver 238, memory240, a display driver 242 and a temperature sensor 244. The transceiver238 may be configured to send and receive cellular signals, Bluetoothsignals, Wi-Fi signals and geosynchronous signals via an antenna 246 tothe other external devices 234. In various embodiments, applications,modules, data and other information may be stored in the memory 240.

The accelerometer 226 a may be oriented to sense a pitch movement of theframe 204 (e.g., a movement of the user's head, etc.) about a pitch (orhorizontal) axis. The pitch axis may be defined from left to rightthrough the frame 204. The second accelerometer 226 b may be oriented tosense a yaw movement of the frame 204 (e.g., a movement of the user'shead, etc.) about a yaw (or vertical) axis. The yaw axis may be definedas a vertical (relative to the ground) axis through the frame 204.

As a result, each of the accelerometers 226 a and 226 b may beoperational to sense and report a direction of acceleration. Forexample, the pitch accelerometer 228 a may generate a positive value inresponse to a forward rotation (e.g., user's nose moving down, etc.) ofthe frame 204 about the pitch axis. A negative value may be generated inresponse to a backwards rotation (e.g., user's nose moving upwards,etc.) of the frame 204 about the pitch axis. Likewise, the yawaccelerometer 226 b may generate a positive value for a clockwiserotation of the frame 204 (e.g., user's left ear forward, right earbackwards, etc.) about the yaw axis. A negative value may be generatedfor a counterclockwise rotation of the frame (e.g., user's left earbackwards, right ear forward, etc.) about the yaw axis. Thus, an outputsignal from the pitch accelerometer 226 a may be used to answer atelephone call in response to the accelerometer 226 a detecting anodding “yes” (pitch) movement of the user's head. A shaking “no” (yaw)movement of the user's head may cause the second accelerometer 226 b tosend a different command to circuit 216 to not answer the call.

The circuit 216 may be configured to provide two-way radiocommunications with other external devices 234 (e.g., other phones,computers, etc.). For example the electronics/RF circuit 216 may receivetelephone calls or display data on LCD 230 of display 202. Display 202includes LCD 230 and backlight 232. The commands provided to thewearable computer 200 may be generated in a hands-free manner (e.g.,without a user pressing a button with his or her hands, etc.). Thecommunications may be implemented using a wireless Bluetooth standard, awireless Universal Serial Bus (USB) standard, a cellular telephonestandard, a Wi-Fi standard, etc. Other standard and/or proprietyprotocols may be implemented to meet the criteria of a particularapplication.

4. Exemplary Data Display Method

The system of FIG. 1 or the wearable computer of FIG. 2 may beconfigured to perform the method steps of FIG. 4 . Moreover, the methoddescribed in FIG. 4 is an example of an embodiment of present methodsfor displaying dockable billboards for labeling objects in athree-dimensional display of a virtual or real environment. It should beunderstood by reference to this disclosure that the method describes anexample embodiment of the methods steps carried out by the presentsystem, and that other example embodiments may be created by addingother steps or by removing one or more of the method steps described inFIG. 4 .

FIG. 4 depicts an example method for displaying dockable billboards forlabeling objects in a two-dimensional or three-dimensional display of a(e.g., a navigable, etc.) virtual and/or real three-dimensionalenvironment. The method begins at step 300 when the system determines alocation and a camera viewpoint of a virtual camera positioned in thethree-dimensional environment. In various embodiments, the cameraviewpoint of the virtual camera may represent a perspective of a userrepresented at the location in the three-dimensional environment, andthe location of the virtual camera may be the same as the user'slocation as the user is being represented in the three-dimensionalenvironment. In some embodiments, the location and camera viewpoint ofthe virtual camera can be determined by the physical location andorientation as detected by one or more sensors (e.g., motion sensors,location sensors, orientation detection sensors, accelerometers,magnetic sensors, GPS sensors, etc.) worn by a user as the usernavigates around the user's environment. For example, in variousembodiments having a wearable computer attached to the user's head(e.g., via spectacle frames, etc.), a scene captured by the wearablecomputer may be displayed on a display coupled to the wearable computerso that the user can view a scene of the surrounding environment withdata billboards linked to, or overlaid on, at least one of the objectsbeing viewed in the environment. Thus, in this example, the position,location, orientation, motion, etc., of the wearable computer, asdetermined by various sensors attached to the wearable computer, may beused to obtain data associated with the viewed objects. Continuing withthis example, the data associated with the objects may be, but is notlimited only to, real-time streaming data (e.g., data obtained over acellular data connection, a Wi-Fi connection or a Bluetooth connection,etc.) or the data may be stored in memory that is part of the wearablecomputer.

At step 302, the system 100 or system 200 may be configured to determinea distance between the camera (virtual camera) location and at least oneobject located in the environment. In various embodiments, the virtualcamera location and the location of at least one object may becalculated based on a common coordinate system (e.g., Cartesian, polar,etc.) for the virtual environment. In other embodiments that use awearable computer, the location of the virtual camera may be determinedusing geosynchronous information that is received by the wearablecomputer, and the location of each object in the three-dimensionalenvironment may be calculated with respect to the location of thewearable computer based on a known longitudinal and latitudinal positionof each object.

At step 304, the system 100 or wearable computer 200 determines whethera location for the object with respect to the location of the camera(virtual camera) viewpoint satisfies a first criterion (e.g., a datadisplay docking criterion, etc.). In various embodiments, the firstcriterion may be the proximity of the object to the location of thevirtual camera (e.g., the location of the user as represented in thethree-dimensional environment, etc.), where the first criterion issatisfied by a set number of objects that are closest to the location ofthe virtual camera. In other embodiments, the first criterion may be athreshold distance, where the first criterion is satisfied when thedistance between the location of an object (e.g., all-around, in frontof the virtual camera, etc.) and the location of the camera itself iswithin the threshold distance. In still other embodiments, the firstcriterion may be a predefined radius around the location of the camera(virtual camera), where the first criterion is satisfied if an objectlocation is within the predefined radius. It should be understood inlight of this disclosure that there are many ways to determine thedistance between an object in a three-dimensional environment and acamera (virtual camera) represented in the three-dimensionalenvironment, and all such methods are contemplated by the presentdisclosure.

If an object location, with respect to the location of the camera(virtual camera) satisfies the first criterion, at step 306, the system100 or wearable computer 200 moves a data billboard for the object to apredetermined location in (or a designated portion of) a viewing area(which may be a part or the entire display area of a display/screen of acomputing device) of the three-dimensional environment. Thepredetermined location may be any location where one or more datadisplays may be grouped for the objects that satisfy the firstcriterion. In various embodiments, the predetermined location of theviewing area may be away from the object to which the billboardcorresponds, proximate to a specific designated location, or in adesignated area such as an area along a bottom edge, a left edge, topedge, right edge, etc., of the viewing area. In one or more embodiments,the predetermined location may be top or side edge of the viewing area.

It should be understood from reference to this disclosure that invarious embodiments, the viewing area corresponds to a scene (e.g., adisplayed view, etc.) of the three-dimensional environment that would becaptured from a viewpoint positioned at the location and the perspectiveof the virtual camera. If a data billboard for an object is alreadylocated at the predetermined location in the viewing area, in variousembodiments, the system maintains the position of the data billboard atthe predetermined location (e.g., keeps the billboard docked at thebottom of the viewing area, etc.). In various embodiments, when the databillboard is moved to the predetermined location, it may be linked toits corresponding object using a lead line (e.g. a solid or dotted line,etc.). In some embodiments, solid lines may be used to link a billboardto an object that can be seen from the camera viewpoint and dotted linesmay be used to link billboards to objects that are located behindanother object and not in the line of sight, in displayed views of thethree-dimensional environment as perceived from the camera viewpoint. Instill other embodiments, the data billboard may be colored to match acolor of its corresponding object. In still other embodiments, theobject may be marked with a marker (e.g., a shaped marker, an encirclednumber, etc.) and the data billboard may contain or be associated with acorresponding marker thereby allowing the user to link each docked databillboard with its corresponding object.

In embodiments using a wearable computer having a camera, the viewingarea (the displayed view as rendered on the display with the user) ofthe three-dimensional environment is a scene that is captured by thecamera. In still other embodiments that use a heads-up display (e.g., atransparent display associated with a wearable computer, etc.), theviewing area (the displayed view as rendered on the display with theuser) of the three-dimensional environment corresponds to that portionof the real-world environment that may be viewed by the user through theheads-up display or through the transparent display. In theseembodiments, moving the data billboard to a designated area of theviewing area such as a bottom edge, etc., of the viewing area furthercomprises moving the data billboards for those objects that satisfy thefirst criterion to a bottom edge of the heads-up display, the bottomedge of the transparent display or a bottom edge of a two-dimensionaldisplay that displays the viewing area of the three-dimensionalenvironment (which may overlaid or superimposed with a portion of thereal-world three-dimensional environment in some embodiments).

In various embodiments, each data billboard is arranged in thedesignated area so that the data billboards do not overlap with oneanother. This allows the user to read the content of each of the databillboards. In some of these embodiments, the data displays (e.g.,billboards docked or moved into the predetermined location, thedesignated area, etc.) may be presented in the same order as the orderof the objects—from which the data displays were moved into their dockedposition in the predetermined location, the designated area, etc.—indisplayed views of the three-dimensional environment as perceived fromthe location and the perspective of the camera (virtual camera), forexample, about the yaw angle (e.g., horizontally, etc.). In still otherembodiments, at step 308, as the perspective of the camera pans (e.g.,the yaw angle changes, etc.) across the three-dimensional environment,the data billboards for objects that satisfy the first criterion aredynamically rearranged based on lateral changes in the relativepositions of the objects in displayed views of the three-dimensionalenvironment, as perceived by the camera (virtual camera), etc. Themethod then restarts at step 300.

If, on the other hand, a location of an object with respect to thelocation of the camera (virtual camera) viewpoint fails to satisfy thefirst criterion, at step 310, the system 100 or the wearable computer200 removes the data display if docked at the particular location in theviewing area back to the object, or if the data billboard was notdocked, the system maintains the data billboard near or on itscorresponding object. At step 312 the system may rearrange the existingdocked billboards in accordance with at least a second criterion. Thesecond criterion can, but is not limited to, a docked display orderingcriterion that orders lateral positions of data billboards in the sameorder as the order of objects to which the data bill boards correspondin displayed views of the three dimensional environment as perceivedfrom the location and the perspective of the virtual camera. Once thedocked data billboards are rearranged, the method then restarts at step300.

In various embodiments, in response to receiving (e.g., user, system,etc.) input specifying that the camera (virtual camera) changes from afirst location to a second location and/or from a first perspective to asecond perspective, the system dynamically moves each data billboardfrom the predetermined location in the predetermined location,designated area, etc. of the viewing area (e.g., undocks the billboard,etc.) back to an undocked position (e.g., overlaid on an object oradjacent the object to which the data billboard corresponds, etc.) oncethe object corresponding to the data billboard fails to satisfy thefirst criterion. Furthermore, the system dynamically moves eachbillboard for its corresponding object from a position on, or adjacentto, the object to the predetermined location, designated area, etc., inthe viewing area (e.g., docks a data display, etc.) once the objectcorresponding to the billboard satisfies the first criterion. Moreover,in various embodiments, the system may be configured to rearrange thedocked billboards as the camera (virtual camera) viewpoint moves fromthe first location to the second location based on at least a secondcriterion. In some embodiments, the second criterion may be the relativelocation of each object with respect to the other objects in displayedviews of the three-dimensional environment as perceived from the camera(virtual camera).

5. Example User Experiences

The operation of example embodiments of the present system and methodsare perhaps best understood by reviewing particular examples. Thefollowing examples describe the experience of the user while using thedata display system of FIG. 1 or the wearable computer of FIG. 2 . Inthe examples described herein, the three-dimensional environment isshown on a two-dimensional display that is viewed by the user. However,it should be understood from reference to this disclosure that whenusing the wearable computer, the real-world environment may be viewed bythe user through a heads-up display that overlays the billboardsassociated with objects onto the viewed scene in a manner that linkseach billboard to its corresponding object. Thus, in embodiments thatuse a heads-up display, the term heads-up display should be understoodto include instances where the data billboards are projected onto thelenses of glasses, or the lenses of the glasses are configured tooperate as a transparent display so that the billboards are overlaidonto the scene being viewed by the user through the transparent display.Additionally, the data billboards may be directly projected onto theretinas of the user so that the billboards are perceived by the user asbeing projected into the environment.

6. First Example Experience

Referring to FIG. 5 through FIG. 8 , a three-dimensional environment isshown having one or more objects 402-412 located therein. Referringparticularly to FIG. 5 , at least one of the objects has a data display(e.g., data billboard, etc.) 414, 416, 418, 422, and 424 located on itscorresponding object. In the example shown, each object corresponds to aprocessor in a multi-processor environment (e.g., a server farm, etc.).Thus, the dimensions of the object correspond to a facet of dataassociated with the processor. In various embodiments, the data displays414, 416, 418, 422, and 424 display real-time textual data associatedwith the corresponding object (e.g., processor, etc.). Referring to FIG.6 , as the camera (e.g., virtual camera, a user represented by thethree-dimensional environment, etc.) moves from an initial location andan initial perspective corresponding to the displayed view shown in FIG.5 to a first location and a first perspective corresponding to thedisplayed view shown in FIG. 6 , the object 402 moves within aparticular proximity (e.g., satisfying a first criterion as previouslydiscussed, etc.) with respect to the camera causing the data billboardto move (e.g., dock, etc.) to the predetermined location 426 (e.g., adesignated area, etc.) in a viewing area 428 (e.g., a displayed view,etc.) of the virtual environment 400. That is, the viewing area 428 isthat portion of the scene of the (e.g., virtual, etc.) three-dimensionalenvironment 400 that is perceived by a virtual camera with the firstlocation and the first perspective in the (e.g., virtual, etc.)three-dimensional environment. In some embodiments, a data display (or adata display area thereof) that corresponds to (e.g., associated with,etc.) a three-dimensional object displays the same content when the datadisplay is undocked (e.g., located at a position aligned with theposition of the three-dimensional object in displayed views of thethree-dimensional environment, etc.) and when the data display is dockedin a predetermined location (e.g., a designated area not aligned withthe position of the three-dimensional object, etc.). A data display asdescribed herein can be used to display textual data as well asnon-textual data. For example, a data display may be used to displaygraphic data, map data, topological data, topographic data, actionablecontrols (e.g., a user selectable control to perform an operation or anaction, etc.), etc. In some embodiments, a data display is interactivein that a user can perform user interaction, perform user input, etc.,with respect to the data display or displayed components therein. Forexample, a user may be presented with high-level depiction of abuilding, a system, etc., initially in a data display; the user canselect an object portrayed in the high-level depiction to find detailedinformation, detailed sub-components, etc., in a component representedby the selected object.

Referring to FIG. 7 , when the camera viewpoint moves from the firstlocation and the first perspective to a second location and a secondperspective in the three-dimensional environment, the viewing area 430of the three-dimensional environment, as perceived from the camera atthe second location with the second perspective, changes as illustratedin the figure. As such, object 402 remains within the particularproximity (e.g., satisfying a first criterion as previously discussed,etc.) with respect to the camera at the second location with the secondperspective (e.g., the billboard 414 remains docked, etc.) but otherobjects 404, 405, 406 and 410 move within the particular proximity withrespect to the camera at the second location with the secondperspective. Thus, the billboards 416, 418 and 422 dock in thepredetermined location 426 (e.g., the designated area, etc.) innon-overlapping positions. In some embodiments, the billboards 416, 418and 422 as docked are arranged in the same order as the order of lateralpositions of the objects to which the billboards 416, 418 and 422respectively correspond in the (e.g., currently, etc.) displayed view ofthe three-dimensional environment (or the viewing area), as perceivedfrom the camera at the second location with the second perspective.Because object 412 remains outside the particular proximity (e.g., notsatisfying the first criterion, etc.) with respect to the camera at thesecond location with the second perspective, billboard 424,corresponding to object 412, remains undocked and positioned on theobject 412.

Referring to the upper viewing area 432 in FIG. 8 , as perceived fromthe camera (or virtual camera) at a third location with a thirdperspective, multiple billboards 412, 414, 416, 418, 420, 422, and 424are docked at the predetermined location 426 (e.g., in the designatedarea, etc.) in viewing area 432. That is, with respect to the thirdlocation and the third perspective, each of the objects 402, 405, 406,408, 410 and 412 are located at a distance within the particularproximity from the camera at the third location with the thirdperspective such that each corresponding billboard 414, 416, 418, 47,420, 422 and 424 is docked at the predetermined location 426 (e.g., inthe designated area, etc.) of the viewing area 432. Moreover, eachdocked billboard is arranged according to the relative lateral positionof its corresponding object with respect to the other objects, asperceived from the camera at the third location with the thirdperspective.

Referring to the lower viewing area 434, as perceived from the camera(or virtual camera) when the camera is moved to a fourth location and afourth perspective, all of the same objects shown in the upper viewingarea 432 are still within the particular proximity to the camera at thefourth location with the fourth perspective. Thus, all of the dockedbillboards remain docked. However, in the viewing area 434 as perceivedfrom the camera at the fourth location with the fourth perspective, thelateral position of each object has changed with respect to the otherobjects. As a result, billboards 417 and 418 are dynamically reordered,and billboards 422 and 424 are dynamically reordered as the camera movedfrom the third location and the third perspective to the fourth locationand the fourth perspective.

In all of the viewing areas shown in FIG. 5 through FIG. 8 , the datapresented on the billboards may be real-time textual data, non-real-timetextual data, etc., associated with a processor that corresponds to theobject associated with the respective billboard.

7. Second Example User Experience

FIG. 9 and FIG. 10 illustrate a three-dimensional environment 436 havingone or more objects 438-448 located therein. In the particularembodiment shown in FIG. 9 and FIG. 10 , the three-dimensionalenvironment is being viewed using a wearable computer or directly by theuser through a heads-up display. In some embodiments, the user may bewearing a computer coupled to a camera where images captured by thecamera of the real-world environment surrounding the user are viewed ona two-dimensional display. In some embodiments, the user may be viewingthe real-world environment surrounding the user through a transparentheads-up display or transparent display where billboards containing dataassociated with the objects being viewed are overlaid or linked to theobjects. For example, the real-world environment may be a town or city(e.g., New York City, etc.) and the objects in the user's view may be aretail store (e.g., a coffee shop, a building supply store, etc.), aservice provider (e.g., a doctor, a dentist, etc.), etc., that may belocated in one or more buildings. The objects being viewed in thereal-world environment may also be a landmark (e.g., the Empire Statebuilding, a museum, a monument, etc.).

Referring particularly to FIG. 9 , a user (with which a camera orvirtual camera is collocated and shares the same location andperspective) is standing on 5th avenue 450 in New York City wearing thewearable computer of FIG. 2 . At least one object has an associated databillboard 452, 454, 456, 458, 460, or 462 located proximate acorresponding object. Each of the data billboards may be linked to arespective object of the objects via a line or other suitable indicatorsuch as an arrow, as shown in the figure. In various embodiments,linking a billboard to its corresponding object may also be carried outby, for example, color coding the billboard and its corresponding object(e.g., with the same color, with the same brightness, with the sametexture, etc.).

In the embodiment of FIG. 9 , the viewing area 464 is displayed on adisplay such as 202 of FIG. 3 , etc., of a computing device such as thewearable computer 200, etc. In other embodiments, the viewing area 464may also be the scene actually viewed by the user's eyes through aheads-up display (not shown) or through a transparent display (e.g., inthe shape of lenses contained in a glass frame (not shown), etc.) wherethe billboards 452, 454, 456, 458, 460, and 462 are overlaid on theviewing area based on location information computed by the wearablecomputer 200 using data (e.g., GPS data, magnetic compass signals, etc.)obtained from one or more of the sensors that are part of the wearablecomputer device 200.

In various embodiments, the billboards 452, 454, 456, 458, 460, and 462contain data (e.g., textual, hyperlinks, pictures, etc.) associated withthe corresponding object (e.g., business establishment, etc.). Based ona camera viewpoint location shown in FIG. 9 , the retail establishmentrepresented by object 438 satisfies at least one criterion (e.g., iswithin a certain proximity to the location of the camera viewpoint,etc.). As a result, the billboard 452 is moved to a predefined location466 (e.g., a designated area, etc.) in the viewing area 464. In someembodiments, the predefined location 466 is positioned proximate thebottom edge of the viewing area 464. In other embodiments, the viewingarea need not be predefined and the system may dynamically choose thearea that the billboards are docked based on what is being viewed (e.g.,by choosing an area that is least obstructive to the user's view, etc.).

In various embodiments, the at least one criterion may be defined as apreset number (e.g., five, etc.) of objects that are closest to thelocation of the camera viewpoint. In other embodiments, the at least onecriterion may be satisfied by any object whose distance from the cameraviewpoint location is less than a threshold distance. In still otherembodiments, the at least one criterion may be satisfied when an objectis located within a defined radius from the location of the cameraviewpoint.

Referring to FIG. 10 , as the camera (e.g., with the location and theperspective of the user, etc.) moves from a first location and a firstperspective (e.g., corresponding to the user's location and perspectivein FIG. 9 , etc.) to a second location and a second perspective (e.g.,corresponding to the user's location and perspective in FIG. 10 , etc.),the objects 438, 440, 442, 444 and 446 move with respect to one another,as perceived from the camera in the second position with the secondperspective. Thus, when the camera is at the second location with thesecond perspective, the distances between the location of the camera andthe location of objects 438, 440, 442, and 444 satisfy the firstcriterion, thereby resulting in the billboards that corresponds to theseobjects being docked at the bottom of the viewing area 466 in an orderfrom left to right that corresponds to the order of the objects fromleft to right, as perceived from the camera at the second location withthe second perspective.

In some embodiments, the same three-dimensional environment as describedherein can be explored by multiple users represented at the same or evendifferent locations in the three-dimensional environment. For example,the three-dimensional environment may be an environment that representsa first user in Chicago and a second user in San Francisco.

The first user and the second user can have their respectiveperspectives at their respective locations. The first user and thesecond user can have their own displayed views of the samethree-dimensional environment on their own computing devices. At theirchoosing, the first user and the second user can explore a portion ofthe three-dimensional environment in a collaborative ornon-collaborative manner; exchange their locations or perspectives;exchange messages/information/history with each other; etc.

A history of a user's location and/or the user's perspective asgenerated by the user's exploration (e.g., via the control of a virtualcamera representing the user's location and perspective, etc.) in athree-dimensional environment as described herein may constitute atrajectory comprising one or more time points and one or more ofuser-specified waypoints, system-generated waypoints, user-specifiedcontinuous spatial segments, system-generated continuous spatialsegments, as traversed by the user in the three-dimensional environmentat the respective time points. The trajectory of the user in thethree-dimensional environment can be recorded, replayed (or playedback), paused, rewound, fast-forwarded, altered, etc.

A history of underlying data that supports a user's exploration (e.g.,via the control of a virtual camera representing the user's location andperspective, etc.) in a three-dimensional environment as describedherein may be recorded by a system as described herein. Instead ofplaying back the user's own history of exploration, the underlying datathat supports the user's particular exploration can be explored orre-explored with same or different locations and/or perspectives ascompared with those of the user's own history of exploration.

8. Example Process Flow

FIG. 12 illustrates an example process flow. In some embodiments, thisprocess flow is performed by a data display system (e.g., as shown inFIG. 1 , etc.) comprising one or more computing devices or units. Inblock 1202, the data display system causes a first view of athree-dimensional environment to be displayed to a user at a first time.The first view of the three-dimensional environment is a view of thethree-dimensional environment relative to a first location and a firstperspective of the user as represented in the three-dimensionalenvironment at the first time. The first view comprises a data displaywith a data display area aligned with an object area of thethree-dimensional object. The three-dimensional object does not satisfya data display docking criterion.

In block 1204, the data display system receives user input thatspecifies that the user as represented in the three-dimensionalenvironment has relocated in the three-dimensional environment and isnow at a second location and a second perspective. A combination of thesecond location and the second perspective is different from acombination of the first location and the first perspective.

In block 1206, the data display system, in response to receiving theuser input, determines whether the three-dimensional object satisfiesthe data display docking criterion.

In block 1208, the data display system, in response to determining thatthe three-dimensional object satisfies the data display dockingcriterion, causes a second different view of the three-dimensionalenvironment to be displayed to a user at a second time later than thefirst time. The second view is a view of the three-dimensionalenvironment relative to the second location and the second perspectiveof the user as represented in the three-dimensional environment at thesecond time. The data display area of the data display in the secondview is relocated to a designated area that does not align with theobject area of the three-dimensional object.

In an embodiment, a second three-dimensional object is represented inthe three-dimensional environment; the first view comprises a seconddata display visually aligned with the second three-dimensional object.The data display system can be configured to perform: in response toreceiving the user input, performing: determining whether the secondthree-dimensional object, as viewed from the second location and thesecond perspective, satisfies the data display docking criterion; inresponse to determining that the second three-dimensional object, asviewed from the second location and the second perspective, does notsatisfy the data display docking criterion, maintaining the second datadisplay in a position visually aligned with the second three-dimensionalobject in the second view.

In an embodiment, a second three-dimensional object is represented inthe three-dimensional environment; the first view comprises a seconddata display, for the second three-dimensional object, in the predefinedarea. The data display system can be configured to perform: in responseto receiving the user input, determining whether the secondthree-dimensional object, as viewed from the second location and thesecond perspective, satisfies the data display docking criterion; inresponse to determining that the second three-dimensional object, asviewed from the second location and the second perspective, satisfiesthe data display docking criterion, maintaining the second data displayin the predefined area in the second view.

In an embodiment, a second three-dimensional object is represented inthe three-dimensional environment; the first view comprises a seconddata display, for the second three-dimensional object, in the predefinedarea. The data display system can be configured to perform: in responseto receiving the user input, determining whether the secondthree-dimensional object, as viewed from the second location and thesecond perspective, satisfies the data display docking criterion; inresponse to determining that the second three-dimensional object, asviewed from the second location and the second perspective, does notsatisfy the data display docking criterion, moving the second datadisplay away from the predefined area to a second location visuallyaligned with the second three-dimensional object in the second view.

In an embodiment, the designated area comprises a plurality of dockeddisplay areas away from a plurality of three-dimensional objects towhich the plurality of docked display areas corresponds; the pluralityof docked display areas is displayed in the designated area in an orderbased on a docked display ordering criterion.

In an embodiment, the designated area comprises a plurality of dockeddisplay areas away from a plurality of three-dimensional objects towhich the plurality of docked display areas corresponds; the pluralityof docked display areas is displayed in the designated area in an orderbased on a lateral order of the plurality of three-dimensional objectsalong a spatial direction in the three-dimensional environment.

In an embodiment, the designated area comprises a plurality of dockeddisplay areas non-overlapping with one another in displayed views of thethree-dimensional environment.

In an embodiment, the data display system is further configured to, inresponse to receiving input specifying changes in the user's locationand perspective as represented in the three-dimensional environment,dynamically order a plurality of docked display areas in the designatedarea in a plurality of displayed views of the three-dimensionalenvironment based on a docked display ordering criterion.

In an embodiment, content displayed in the data display area relocatedto the designated area in the second view is the same as contentdisplayed in the data display area aligned with the object area of thethree-dimensional object in the first view.

In an embodiment, content displayed in the data display area relocatedto the designated area in the second view is different from contentdisplayed in the data display area aligned with the object area of thethree-dimensional object in the first view.

In an embodiment, the designated area, as located in a display used torender views of the three-dimensional environment to the user, does notvary with the user's location and perspective as represented in thethree-dimensional environment.

In an embodiment, the data display area is aligned in the first viewwith one of an adjacent area position, a left area portion, a right areaportion, a top area portion, a bottom area portion or a center areaportion of the three-dimensional object as represented in the firstview.

In an embodiment, a three-dimensional object and the user arerepresented in the three-dimensional environment with a first distancebetween each other at the first time; and wherein the user is located ata second finite distance to the three-dimensional object in thethree-dimensional environment at the second time.

In an embodiment, the data display docking criterion is based on one ormore of a distance of a three-dimensional object relative to the user'slocation as represented in the three-dimensional environment, an angleof a three-dimensional object relative to the user's perspective asrepresented in the three-dimensional environment, or a configurabledesignation of the three-dimensional object.

In an embodiment, the data display docking criterion is based on adistance of a three-dimensional object relative to the user's locationas represented in the three-dimensional environment.

In an embodiment, the data display comprises content from one or more ofreal-time streaming data, non-real-time streaming data, etc.

In an embodiment, the first view and the second view are rendered on oneof a display of a computing device, a display of a wearable computer, aheads-up display, a display glass, etc.

In an embodiment, the user's location and perspective as represented inthe three-dimensional environment are determined based on one or more ofthe user's actions relating to one or more user input devices, theuser's physical position and physical perspective as detected based onone or more sensors, etc.

In an embodiment, the user's location and perspective as represented inthe three-dimensional environment are determined based on the user'sphysical position and physical perspective as detected based on one ormore of an accelerometer, a geosynchronous positioning sensor, ageomagnetic field sensor, an orientation sensor, a gyroscope, a gravitysensor, a rotation vector sensor, etc.

In an embodiment, the three-dimensional environment is dynamicallysuperimposed with a portion of a real-world three-dimensionalenvironment in which the user moves, and wherein the user input isgenerated through one or more sensors configured to track the user'smotion.

In an embodiment, the invention is directed to a computer-implementedmethod of displaying dockable data displays for labeling objects in athree-dimensional display of a virtual or real environment. Thecomputer-implemented method comprises determining a distance between avirtual camera location within the virtual environment and at least oneobject located in the virtual environment, determining that the at leastone object satisfies a first criterion and at least partially inresponse to determining that the at least one object satisfies the firstcriterion, docking the data display associated with the at least oneobject to a predefined location in a view of the virtual environment, asperceived by the virtual camera.

In another embodiment, the invention is directed to acomputer-implemented method of displaying dockable billboards forlabeling objects in a three-dimensional display of a virtual or realenvironment. The computer-implemented method comprises determining alocation of a camera viewpoint in the environment, where the cameraviewpoint represents the location of a user navigating through theenvironment, determining a distance between the camera viewpointlocation and one or more objects in the environment, and at leastpartially in response to the distance between the camera viewpointlocation and each one of the one or more objects being less than athreshold distance, moving a data billboard, for each one of the one ormore objects whose distance from the camera viewpoint location is lessthan the threshold distance, to a designated location of a viewing area(which may be a part or the entire display area of a display/screen of acomputing device) of the environment as perceived from the location ofthe camera and its viewpoint.

In an embodiment, an apparatus comprises a processor and is configuredto perform any of the foregoing methods.

In an embodiment, a non-transitory computer readable storage medium,storing software instructions, which when executed by one or moreprocessors cause performance of any of the foregoing methods.

In an embodiment, a computing device comprising one or more processorsand one or more storage media storing a set of instructions which, whenexecuted by the one or more processors, cause performance of any of theforegoing methods. Note that, although separate embodiments arediscussed herein, any combination of embodiments and/or partialembodiments discussed herein may be combined to form furtherembodiments.

9. Example Computer Architecture

FIG. 11 illustrates a diagrammatic representation of a computer 500 thatcan be used within the system 100 of FIG. 1 , for example, as one of theclient devices 140 or server 115, shown in FIG. 1 .

In particular embodiments, the computer 500 may be connected (e.g.,networked, etc.) to other computers by a LAN, an intranet, an extranet,and/or the Internet. The computer 500 may be a personal computer (PC), atablet PC, a Personal Digital Assistant (PDA), a cellular telephone, aweb appliance, a server, or any computer capable of executing a set ofinstructions (sequential or otherwise) that specify actions to be takenby that computer. Further, while only a single computer is illustrated,the term “computer” shall also be taken to include any collection ofcomputers that individually or jointly execute a set (or multiple sets)of instructions to perform any one or more of the methodologiesdiscussed herein.

The example computer 500 includes a processing device 502, a main memory504 (e.g., read-only memory (ROM), flash memory, dynamic random accessmemory (DRAM) such as synchronous DRAM (SDRAM) or Rambus DRAM (RDRAM),etc.), a static memory 506 (e.g., flash memory, static random accessmemory (SRAM), etc.), and a data storage device 518, which communicatewith each other via a bus 532.

The processing device 502 represents one or more general-purposeprocessing devices such as a microprocessor, a central processing unit,or the like. More particularly, the processing device may be a complexinstruction set computing (CISC) microprocessor, reduced instruction setcomputing (RISC) microprocessor, very long instruction word (VLIW)microprocessor, or processor implementing other instruction sets, orprocessors implementing a combination of instruction sets. Theprocessing device 502 may also be one or more special-purpose processingdevices such as an application specific integrated circuit (ASIC), afield programmable gate array (FPGA), a digital signal processor (DSP),network processor, or the like. The processing device 502 may beconfigured to execute the processing logic 526 for performing variousoperations and steps discussed herein.

The computer 500 may further include a network interface device 508 thatcommunications across network 515. The computer 500 also may include avideo display unit 510 (e.g., a liquid crystal display (LCD) or acathode ray tube (CRT), etc.), an alphanumeric input device 512 (e.g., akeyboard, etc.), a cursor control device 514 (e.g., a mouse, etc.), asignal generation device 516 (e.g., a speaker, etc.), etc.

The data storage device 518 may include a non-transitorycomputer-accessible storage medium 530 (also known as a non-transitorycomputer-readable storage medium or a non-transitory computer-readablemedium) on which is stored one or more sets of instructions (e.g.,software 522, etc.) embodying any one or more of the methodologies orfunctions described herein. The software 522 may also reside, completelyor at least partially, within the main memory 504 and/or within theprocessing device 502 during execution thereof by the computer 500, themain memory 504 and the processing device 502 also constitutingcomputer-accessible storage media. The software 522 may further betransmitted or received over a network 520 via the network interfacedevice 508.

While the computer-accessible storage medium 530 is shown in an exampleembodiment to be a single medium, the term “computer-accessible storagemedium” should be understood to include a single medium or multiplemedia (e.g., a centralized or distributed database, and/or associatedcaches and servers, etc.) that store the one or more sets ofinstructions. The term “computer-accessible storage medium” shall alsobe understood to include any medium that is capable of storing, encodingor carrying a set of instructions for execution by the computer and thatcause the computer to perform any one or more of the methodologies ofthe present invention. The term “computer-accessible storage medium”shall accordingly be understood to include, but not be limited to,solid-state memories, optical and magnetic media, etc.

10. Equivalents, Extensions, Alternatives and Mischellaneous

In the foregoing specification, embodiments of the invention have beendescribed with reference to numerous specific details that may vary fromimplementation to implementation. Thus, the sole and exclusive indicatorof what is the invention, and is intended by the applicants to be theinvention, is the set of claims that issue from this application, in thespecific form in which such claims issue, including any subsequentcorrection. Any definitions expressly set forth herein for termscontained in such claims shall govern the meaning of such terms as usedin the claims. Hence, no limitation, element, property, feature,advantage or attribute that is not expressly recited in a claim shouldlimit the scope of such claim in any way. The specification and drawingsare, accordingly, to be regarded in an illustrative rather than arestrictive sense.

The invention claimed is:
 1. A computer-implemented method, comprising:rendering a plurality of three-dimensional objects in athree-dimensional environment based on user navigational input, wherein,for each three-dimensional object included in the plurality ofthree-dimensional objects, a corresponding data display area appearsproximate to the three-dimensional object in the three-dimensionalenvironment; detecting a change in user perspective; in response to thechange in user perspective, determining that each of two or morethree-dimensional objects included in the plurality of three-dimensionalobjects is within a specified angle of the user perspective and that adistance between each of the two or more three-dimensional objects andthe user perspective is less than a threshold distance, and responsiveto the determining, moving two or more data display areas respectivelycorresponding to each of the two or more three-dimensional objects fromappearing proximate to the respective two or more three-dimensionalobjects to a foreground area of the three-dimensional environment whiledisplaying, for a third three-dimensional object included in theplurality of three-dimensional objects, a third data display areaproximate to the third three-dimensional object in the three-dimensionalenvironment; and rearranging positions of the two or more data displayareas located in the foreground area, wherein the rearranged positionsreflect the change in user perspective.
 2. The computer-implementedmethod of claim 1, wherein, for a given data display area correspondingto a given three-dimensional object, the user perspective and relativelateral positions of two or more three-dimensional objects included inthe plurality of three-dimensional objects indicate a display order ofthe two or more three-dimensional objects included in the plurality ofthree-dimensional objects as viewed from the user perspective, and aforeground location within the foreground area for the data display areacorresponding to the given three-dimensional object is determined basedon the display order.
 3. The computer-implemented method of claim 1,wherein, for a given data display area corresponding to a giventhree-dimensional object, a foreground location within the foregroundarea for the data display area corresponding to the giventhree-dimensional object is based on an angle between the giventhree-dimensional object and the user perspective.
 4. Thecomputer-implemented method of claim 1, wherein the data display areacorresponding to each three-dimensional object included in the pluralityof three-dimensional objects are non-overlapping with one another in theforeground area of the three-dimensional environment.
 5. Thecomputer-implemented method of claim 1, wherein the data display areacorresponding to each three-dimensional object included in the pluralityof three-dimensional objects comprise content from one or more of:real-time streaming data or non-real-time streaming data.
 6. Thecomputer-implemented method of claim 1, wherein the data display areas,when appearing proximate to each three-dimensional object included inthe plurality of three-dimensional objects, are aligned with one of: anarea adjacent to the three-dimensional object, an area left of thethree-dimensional object, an area right of the three-dimensional object,an area above the three-dimensional object, an area below thethree-dimensional object, or an area centered on the three-dimensionalobject.
 7. The computer-implemented method of claim 1, wherein contentdisplayed in each data display area in the foreground area is thecontent displayed in the data display area that appears proximate toeach corresponding three-dimensional object included in the plurality ofthree-dimensional objects.
 8. The computer-implemented method of claim1, wherein content displayed in each data display area in the foregroundarea is different than content displayed in the data display area thatappears proximate to each corresponding three-dimensional objectincluded in the plurality of three-dimensional objects.
 9. Thecomputer-implemented method of claim 1, wherein the plurality ofthree-dimensional objects is rendered on one or more of: a display of acomputing device, a display of a wearable computer, a heads-up display,or a display glass.
 10. The computer-implemented method of claim 1,wherein: the user perspective corresponds to a position along a path inthe three-dimensional environment in which the plurality ofthree-dimensional objects is rendered, and the position along the pathand the user perspective are determined based on one or more of: auser's actions relating to one or more user input devices, or a user'sphysical position and physical perspective, as detected based on one ormore sensors.
 11. The computer-implemented method of claim 1, wherein:the user perspective corresponds to a position along a path in thethree-dimensional environment in which the plurality ofthree-dimensional objects is rendered, and the position along the pathand the user perspective are determined based on a user's physicalposition and physical perspective, as detected based on one or more of:an accelerometer, a geosynchronous positioning sensor, a geomagneticfield sensor, an orientation sensor, a gyroscope, a gravity sensor, or arotation vector sensor.
 12. The computer-implemented method of claim 1,wherein: the user perspective corresponds to a position along a path inthe three-dimensional environment in which the plurality ofthree-dimensional objects is rendered, the three-dimensional environmentis dynamically superimposed with a portion of a real-worldthree-dimensional environment in which a user moves, and the usernavigational input is generated based on one or more sensors configuredto track a motion of the user within the real-world three-dimensionalenvironment.
 13. One or more non-transitory computer readable storagemedia storing software instructions that, when executed by one or moreprocessors, cause the one or more processors to perform the steps of:rendering a plurality of three-dimensional objects in athree-dimensional environment based on user navigational input, wherein,for each three-dimensional object included in the plurality ofthree-dimensional objects, a corresponding data display area appearsproximate to the three-dimensional object in the three-dimensionalenvironment; detecting a change in user perspective; in response to thechange in user perspective, determining that each of two or morethree-dimensional objects included in the plurality of three-dimensionalobjects is within a specified angle of the user perspective and that adistance between each of the two or more three-dimensional objects andthe user perspective is less than a threshold distance, and responsiveto the determining, moving two or more data display areas respectivelycorresponding to each of the two or more three-dimensional objects fromappearing proximate to the respective two or more three-dimensionalobjects to a foreground area of the three-dimensional environment whiledisplaying, for a third three-dimensional object included in theplurality of three-dimensional objects, a third data display areaproximate to the third three-dimensional object in the three-dimensionalenvironment; and rearranging positions of the two or more data displayareas located in the foreground area, wherein the rearranged positionsreflect the change in user perspective.
 14. The one or morenon-transitory computer readable media of claim 13, wherein, for a givendata display area corresponding to a given three-dimensional object, theuser perspective and relative lateral positions of two or morethree-dimensional objects of the plurality of three-dimensional objectsindicate a display order of the two or more three-dimensional objects ofthe plurality of three-dimensional objects as viewed from the userperspective and a foreground location within the foreground area for thedata display area corresponding to the given three-dimensional object isdetermined based on the display order.
 15. The one or morenon-transitory computer readable media of claim 13, wherein, for a givendata display area corresponding to a given three-dimensional object, aforeground location within the foreground area is based on an anglebetween the given three-dimensional object and the user perspective. 16.The one or more non-transitory computer readable media of claim 13,wherein the data display area corresponding to each three-dimensionalobject included in the plurality of three-dimensional objects arenon-overlapping with one another in the foreground area of thethree-dimensional environment.
 17. The one or more non-transitorycomputer readable media of claim 13, wherein the data display areacorresponding to each three-dimensional object included in the pluralityof three-dimensional objects comprise content from one or more of:real-time streaming data or non-real-time streaming data.
 18. The one ormore non-transitory computer readable media of claim 13, wherein thedata display areas, when appearing proximate to each three-dimensionalobject included in the plurality of three-dimensional objects, arealigned with one of: an area adjacent to the three-dimensional object,an area left of the three-dimensional object, an area right of thethree-dimensional object, an area above the three-dimensional object, anarea below the three-dimensional object, or an area centered on thethree-dimensional object.
 19. A system comprising: a memory; and aprocessor that is coupled to the memory and that: renders a plurality ofthree-dimensional objects in a three-dimensional environment based onuser navigational input, wherein, for each three-dimensional objectincluded in the plurality of three-dimensional objects, a correspondingdata display area appears proximate to the three-dimensional object inthe three-dimensional environment; detects a change in user perspective;in response to the change in user perspective, determines that each oftwo or more three-dimensional objects included in the plurality ofthree-dimensional objects is within a specified angle of the userperspective and that a distance between each of the two or morethree-dimensional objects and the user perspective is less than athreshold distance, and responsive to the determination, moves two ormore data display areas respectively corresponding to each of the two ormore three-dimensional objects from appearing proximate to therespective two or more three-dimensional objects to a foreground area ofthe three-dimensional environment while displaying, for a thirdthree-dimensional object included in the plurality of three-dimensionalobjects, a third data display area proximate to the thirdthree-dimensional object in the three-dimensional environment; andrearranges positions of the two or more data display areas located inthe foreground area, wherein the rearranged positions reflect the changein user perspective.
 20. The system of claim 19, wherein, for a givendata display area corresponding to a given three-dimensional object, theuser perspective and relative lateral positions of two or morethree-dimensional objects of the plurality of three-dimensional objectsindicate a display order of the two or more three-dimensional objects ofthe plurality of three-dimensional objects as viewed from the userperspective and a foreground location within the foreground area for thedata display area corresponding to the given three-dimensional object isdetermined based on the display order.